Managing dependencies between data center computing and infrastructure

ABSTRACT

Techniques for operating a data center include providing at least one fan coil unit operable to circulate a cooling airflow to a human-occupiable workspace of the data center; providing a plurality of computer racks arranged in one or more rows in the human-occupiable workspace; forming one or more warm air aisles between the one or more rows of the plurality of computer racks that are in fluid communication with an inlet of the fan coil unit through a warm air plenum, and also with an outlet of the fan coil unit through the human-occupiable workspace and the plurality of computer racks arranged in one or more rows; adjusting the associated electrical power density of one or more of the plurality of computer racks; and based on the adjustment, adjusting a characteristic of the data center.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation and claims the benefit under 35U.S.C. § 120 of U.S. application Ser. No. 14/222,097, entitled “ManagingDependencies Between Data Center Computing and Infrastructure,” filedMar. 21, 2014, the entire contents of which are incorporated herein byreference.

TECHNICAL FIELD

This document discusses techniques for managing relationships betweencomputing equipment and data center infrastructure.

BACKGROUND

When discussing the “power” of computers, people typically focus onspeed—so-called processing power. The electrical power consumed by thecomputers gets less attention. But to people who operator computer datacenters—facilities that contain hundreds or thousands (or tens ofthousands) of computers serving requests from remote users—theelectrical power can be every bit as important as the processing power.Each computer can consume several hundred watts—the same as severalfloodlights. Multiplying that total across thousands of computers shouldmake plain that the potential consumption level can be fairly high.

The operating computers convert all of that consumed electricity intoheat. And that heat has to be removed. So operating a data center islike an electrical double whammy—you have to pay once to use theelectricity, and you have to pay again to remove the effects of the useof the electricity (which itself requires more electricity). The effectsof power consumed by the critical load in the data center are thuscompounded when one incorporates all of the ancillary equipment requiredto support the critical load, such as pumps, chillers, and other suchcomponents.

SUMMARY

This disclosure describes systems and methods for managing relationshipsbetween electronic equipment, such as servers, and infrastructureequipment (e.g., cooling, ventilation, power) of a facility thatsupports such equipment. In a general implementation, a method includesproviding at least one fan coil unit operable to circulate a coolingairflow to a human-occupiable workspace of the data center; providing aplurality of computer racks arranged in one or more rows in thehuman-occupiable workspace, the plurality of computer racks including aplurality of rack-mounted computing equipment that generates heatrelative to an electrical power density associated with each of thecomputer racks; forming one or more warm air aisles between the one ormore rows of the plurality of computer racks, the warm air aisles influid communication with an inlet of the fan coil unit through a warmair plenum, and also with an outlet of the fan coil unit through thehuman-occupiable workspace and the plurality of computer racks arrangedin one or more rows; adjusting the associated electrical power densityof one or more of the plurality of computer racks; and based on theadjustment, adjusting a characteristic of at least one of the one ormore warm air aisles, a power supply associated with the plurality ofcomputer racks, or the fan coil unit.

In a first aspect combinable with the general implementation, adjustingthe associated electrical power density of one or more of the pluralityof computer racks includes adjusting a utilization of the rack-mountedcomputing equipment; based on the adjusted utilization, adjusting anamount of electrical power supplied to the rack-mounted computingequipment; and based on the adjusted amount of supplied electricalpower, adjusting an amount of cooling airflow supplied to therack-mounted computing equipment.

In a second aspect combinable with any of the previous aspects,adjusting a characteristic of at least one of the one or more warm airaisles, a power supply associated with the plurality of computer racks,or the fan coil unit includes adjusting a characteristic of the one ormore warm air aisles, wherein the characteristic of the one or more warmair aisles includes at least one of a number of warm air aisles in thedata center; a dimension of at least one of the one or more warm airaisles; or a setpoint temperature of at least one of the one or morewarm air aisles.

In a third aspect combinable with any of the previous aspects, thedimension includes a width of the at least one warm air aisle definedbetween two rows of computer racks.

In a fourth aspect combinable with any of the previous aspects,adjusting a characteristic of at least one of the one or more warm airaisles, a power supply associated with the plurality of computer racks,or the fan coil unit includes adjusting a characteristic of the powersupply associated with the plurality of computer racks, wherein thecharacteristic of the power supply includes an operating current.

In a fifth aspect combinable with any of the previous aspects, adjustinga characteristic of at least one of the one or more warm air aisles, apower supply associated with the plurality of computer racks, or the fancoil unit includes adjusting a characteristic of the fan coil unit,wherein the characteristic of the fan coil unit includes a quantity ofthe fan coil unit in the data center; a cooling capacity of the fan coilunit in the data center; a temperature of the cooling air supplied fromthe fan coil unit in the data center; and a temperature of a coolingfluid supplied to the fan coil unit in the data center.

A sixth aspect combinable with any of the previous aspects furtherincludes further adjusting the associated electrical power density ofone or more of the plurality of computer racks; and based on the furtheradjustment, adjusting another characteristic of at least one of the oneor more warm air aisles, a power supply associated with the plurality ofcomputer racks, or the fan coil unit.

A seventh aspect combinable with any of the previous aspects furtherincludes adding an additional plurality of computer racks to the datacenter, the additional plurality of computer racks arranged in one ormore rows in the human-occupiable workspace; and further adjusting theassociated electrical power density of one or more of the plurality ofcomputer racks.

An eighth aspect combinable with any of the previous aspects furtherincludes based on the further adjustment, adjusting anothercharacteristic of at least one of the one or more warm air aisles, apower supply associated with the plurality of computer racks, or the fancoil unit.

In a ninth aspect combinable with any of the previous aspects, adjustinganother characteristic of at least one of the one or more warm airaisles, a power supply associated with the plurality of computer racks,or the fan coil unit includes adding at least one additional warm airaisle into the data center between the rows of the additional pluralityof computer racks.

A tenth aspect combinable with any of the previous aspects furtherincludes based on the additional plurality of computer racks, reducingthe associated electrical power density of the plurality of computerracks in the data center; and based on reducing the associatedelectrical power density of the plurality of computer racks in the datacenter, reducing a dimension of the one or more warm air aisles in thedata center.

In an eleventh aspect combinable with any of the previous aspects, eachwarm air aisle includes a ducted airway that fluidly connects backs ofthe plurality of computer racks that are open to the warm air aisle tothe warm air plenum.

In a twelfth aspect combinable with any of the previous aspects, thewarm air plenum includes an attic space in the data center.

In another general implementation, a data center includes a plurality offan coil units operable to circulate a cooling airflow to ahuman-occupiable workspace of the data center; a plurality of computerracks arranged in one or more rows in the human-occupiable workspace,the plurality of computer racks including a plurality of rack-mountedcomputing equipment that generates heat relative to an electrical powerdensity associated with each of the racks, each of the plurality ofcomputer racks including an adjustable associated electrical powerdensity; and one or more warm air aisles arranged between the one ormore rows of the plurality of computer racks, the warm air aisles influid communication with inlets of the fan coil units through a warm airplenum, and also with outlets of the fan coil units through thehuman-occupiable workspace and the plurality of computer racks arrangedin one or more rows, at least one of the one or more warm air aisles, apower supply associated with the plurality of computer racks, or theplurality of fan coil units including an adjustable characteristic basedon the adjustable associated electrical power density.

In a first aspect combinable with the general implementation, theadjustable associated electrical power density includes utilization ofthe rack-mounted computing equipment; an amount of electrical powersupplied to the rack-mounted computing equipment; and an amount ofcooling airflow supplied to the rack-mounted computing equipment.

In a second aspect combinable with any of the previous aspects, theadjustable characteristic includes at least one of a number of warm airaisles in the data center; a dimension of at least one of the one ormore warm air aisles; or a setpoint temperature of at least one of theone or more warm air aisles.

In a third aspect combinable with any of the previous aspects, thedimension includes a width of the at least one warm air aisle definedbetween two rows of computer racks.

In a fourth aspect combinable with any of the previous aspects, theadjustable characteristic includes an operating current of the powersupply.

In a fifth aspect combinable with any of the previous aspects, theadjustable characteristic includes at least one of a number of theplurality of fan coil units in the data center; a cooling capacity of atleast one of the plurality of fan coil units in the data center; atemperature of the cooling air supplied from at least one of theplurality of fan coil units in the data center; and a temperature of acooling fluid supplied to at least one of the plurality of fan coilunits in the data center.

A sixth aspect combinable with any of the previous aspects furtherincludes an additional plurality of computer racks arranged in one ormore rows in the human-occupiable workspace.

In a seventh aspect combinable with any of the previous aspects, atleast one of the one or more warm air aisles, the power supplyassociated with the plurality of computer racks, or the plurality of fancoil units includes an additional adjustable characteristic based on theadditional plurality of computer racks.

In an eighth aspect combinable with any of the previous aspects, theadditional adjustable characteristic includes at least one additionalwarm air aisle into the data center between the rows of the additionalplurality of computer racks.

In a ninth aspect combinable with any of the previous aspects, theassociated electrical power density of the plurality of computer racksin the data center includes a reduced associated electrical powerdensity based on the additional plurality of computer racks, and adimension of the one or more warm air aisles in the data center includesa reduced dimension based on the reduced associated electrical powerdensity.

In a tenth aspect combinable with any of the previous aspects, each warmair aisle includes a ducted airway that fluidly connects backs of theplurality of computer racks that are open to the warm air aisle to thewarm air plenum.

In an eleventh aspect combinable with any of the previous aspects, thewarm air plenum includes an attic space in the data center.

Various implementations described herein may include one, some, all, ornone of the following features. For example, implementations may providedata center operators with flexibility in layout, and can provide forhigh volume heat removal using relatively simple and inexpensiveequipment. In addition, in certain implementations, much of theequipment can be pre-fabricated and tested at a factory, and thenquickly installed and commissioned on-site, so as to provide for faster“go live” time for a facility, and to allow for less expensive, but moredependable, equipment construction.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

DESCRIPTION OF DRAWINGS

FIG. 1A shows a schematic side view of an example data center and a datacenter cooling system.

FIGS. 1B-1D show schematic top views of the example data center and datacenter cooling system at various points in time and based on one or morevariable adjustments.

FIG. 2A shows a schematic side view of another example data center anddata center cooling system.

FIG. 2B shows a schematic side view of another example data center anddata center cooling system.

FIG. 3 shows a schematic side view of another example data center anddata center cooling system.

FIG. 4A shows a schematic top view of another example data center anddata center cooling system.

FIG. 4B shows a schematic side view of another example data center anddata center cooling system.

FIG. 5 shows an example flow chart for a method of managing data centercooling system.

DETAILED DESCRIPTION

FIG. 1A shows a side perspective view of a data center 100. In general,the data center 100 includes and encloses a plurality of computer racksin a controlled environment. The data center 100 includes infrastructureequipment, described in more detail below, that maintains the datacenter 100, and the computer racks, at ambient conditions suitable foroperation, including cooling equipment and power supply equipment.

As illustrated, the data center 100 receives an outside airflow 130,through louvers 135. The outside airflow 130 is processed by the fancoil unit 110, which generates supply airflow 180 (e.g., a coolingairflow) for a workspace 175. The fan coil unit 110 includes a filter125, a cooling coil 120, and a fan 115. Generally, the filter 125filters the received outside airflow 130. The cooling coil 120 receivesa cooling fluid (e.g., chilled fluid from a chiller or other mechanicalrefrigeration device, chilled glycol, an evaporatively-cooled fluid,cool fluid from a body of water, refrigerant liquid, or otherwise) at acooling fluid inlet 152 and lowers the temperature of an airflow thatenters the fan coil unit 110. In some implementations, the cooling coil120 may be a combination of multiple cooling coils, a direct evaporativecooling module, an indirect evaporative cooling module, or combinationthereof.

Cooling fluid can be circulated to the inlet 152 by a pump (or bynatural circulation, for example, in a direct expansion system). Thepump can take any appropriate form, such as a standard centrifugal pump.The cooling coil 120 can have a large surface area and be very thin soas to present a low pressure drop to the data center 100. In this way,slower, smaller, and quieter fans 115 can be used to drive air throughthe fan coil unit 110. As the air passes through the coil 120, its heatis transferred into the fluid in the coil 120, and the air that entersthe fan coil unit 110 is cooled.

The fan 115 can circulate the generated supply airflow 180 to theworkspace 175. The supply airflow 180 is circulated through open facesof the racks 145 and over servers 150 (e.g., server trays) that includeheat-generating electronic devices (e.g., processors, memory, andotherwise). As the supply airflow 180 is circulated through the racks145 and over the servers 150, heat is transferred from the servers 150to the supply airflow 180. In some implementations, an amount of heatgenerated by the servers 150 and transferred to the airflow 180 may berelated to, for example, a temperature of the airflow 180 relative to atemperature of the devices, a flowrate of the airflow 180, and a powerdensity of the servers 160.

Server power density, in some implementations, may be a metric of thepeak power draw per spatial unit by the rack-mounted computers (e.g.,servers 150) in the data center 100. The power density may set thepower-carrying capacity requirement of the power grid. In particular,the power density can be given in units of power per length, e.g.,kilowatts per foot (kw/ft). This may be a useful metric, because it setsthe power-carrying the density of power taps along a power delivery lineand the power capacity of the power taps (e.g., connections to theservers 150). In particular, the number of taps per unit lengthmultiplied by the amperage of those taps cannot exceed a maximum powerdensity. The power density may also set the power capacity requirementsof the power supply cabling, e.g., the circuits, from a power supplymodule 185 that is connected to a power supply 190. The maximum powercapacity requirements of the power distribution lines, e.g., thebusbars, can be somewhat lower (on a per circuit basis, not on anabsolute basis) than the circuits, since averaging discussed below willlower the peak power draw. The capacity of the physical components,e.g., the busbars, plugs, switching gear, etc., that compose a line maybe predetermined (e.g., are often given by manufacturer'sspecifications). Since the power cabling and power distribution lines donot need to be overdesigned, wasted infrastructure cost may be avoided.

For example, assuming that a row 140 is to include two racks 145, eachrack is 19″ wide and is to hold twenty-five severs 150, and each futureserver 150 is projected to consume 400 watts of power at peak operation,then the design power density can be calculated as (400watts/computer*25 computers/rack*2 racks/19″ *12″/foot)≈12.6 kw/ft.

An oversubscription ratio may be determined for the data center 100based on a ratio of design power density to an average power density forpresently available servers 150. A design power density is based on theprojected peak power draw of a future server 150. However, the averagepower draw is usually lower, and often significantly lower than the peakpower draw. There are several factors that contribute to this effect.First, as noted above, the power draw of computers is expected toincrease, and thus the power draw for future computers is set higherthan the power draw of presently available computers. Second, the peakpower draw is greater than the average power draw. In particular, theindividual components (e.g., circuits) that carry power need to be ableto handle the peak power draw of a computer. However, in the aggregateof thousands of computers, statistically there is a much smallerdeviation from the average power draw.

Another way to consider the oversubscription ratio is as ratio of peakpower draw over the life of the data center 100 at the server level tothe peak power draw (per presently available computers) at the facilitylevel. Computer utilization may be substituted as a measure of powerdraw. Thus, the oversubscription ratio can be based on the ratio of peakutilization at the computer level to peak utilization at the facilitylevel.

Inputs for determining the oversubscription ratio include measuredplatform data, measured application data and “forklift” data. Forkliftdata is the information regarding a target number of platforms, andtheir type, to move into a facility, such as data center 100.

Initially, a power usage data can be collected experimentally. A powerdistribution function (percentage time spent at or below a givenfraction of peak power) can be determined experimentally at a variety oflevels of the power distribution system (e.g., at the server, rack, row,or facility) for a variety of platforms (e.g., each platform is aparticular model of server 150 with a particular processing speed,available memory, etc.) for a variety of applications (e.g., searchrequests, email, map requests). In particular, a group of similarplatforms can be run using the same application(s). During this time,the power usage can be measured at the server level, the rack level,and/or the facility level (or other levels). The measurements areperformed for sufficient time to obtain a statically significantlysample of power, e.g., for 24 hours. Average and peak power draw can becalculated from the power distribution function. A cluster can beconsidered to be a large number of computers, e.g., five to ten thousandcomputer, that are logically connected, e.g., tasked to perform the sameapplication, e.g., serving search requests. There can be more than onecluster in a facility.

In addition, much simpler baseline power usage data can be collected forparticular computers that are proposed to be installed in the facility.This baseline power usage data can be a power distribution function forthe particular computer as determined experimentally by running astandard application, a measured average or peak power usage asdetermined experimentally by running a standard application, or simply anameplate power for collected from manufacturer specifications. Wherepower usage is determined experimentally, the application can be a highdemand application.

The power usage data can be entered into power planning software. Thepower planning software can include a power usage database that storesthe data, and a power calculation algorithm. The algorithm canextrapolate the expected power usage from a baseline for a proposedcomputer based on the relationships between known power distributionfunctions. For example, once a power distribution function is measuredfor a proposed new server 150 running a standard search service, thenthe peak power usage for a cluster of such servers 150 running such asearch service can be extrapolated using the relationship between themeasured power distribution functions at the server and cluster levelfor other servers running the search service.

Using the power planning software, an expected average power usage canbe calculated for an exemplary platform and a desired application. Theexemplary platform is selected to be somewhat less demanding thanexpected future platforms, although possibly somewhat more demandingthan a “mid-performance” platform available at the time of design. Giventhe exemplary platform and the desired application, the expected averagepower usage can be determined at the facility level, and anoversubscription ratio calculated.

Once determined, a design power density can be used for multiplefacilities, whereas the oversubscription ratio depends on theapplications to be performed by the servers 150 to be installed in aparticular facility, and thus can vary from facility. Thus, the designpower density tends to be determined first and remain constant forrepetition of the oversubscription step for different facilities.

Once the design power density and oversubscription ratio are determined,spatial planning can be performed for data center 100. Spatial planningincludes determination of size of the facility and layout of the powerdistribution lines and taps. In particular, given the available powerfor the facility (e.g., 1 to 2 megawatts), the total length of therow(s) 140 of racks 145 can be calculated by dividing available power bythe design power density and multiplying by the oversubscription ratio.

For example, assuming that the available power is 1.5 megawatts, thedesign power density is 12.6 kw/ft, and the oversubscription ratio is 2,the total row length will be given by (1.5 megawatts/12.6 kw/ft*2)≈238feet. This could be divided into a single row of 238 feet, two rows of119 feet, three rows of 80 feet, and so on. In contrast, if theoversubscription ratio is 1.0, then the total row length would be 119feet, which could be provided by a single row of 119 feet, two rows of60 feet, etc.

As shown, a return airflow 165 is expelled through a chimney 160 placedadjacent to two server rack rows 140. The return airflow 165, nowcarrying the heat transferred from the servers 150, is circulated into awarm air plenum 170, such as an attic space, located adjacent to theworkspace 175. Return airflow 165 is circulated back to the fan coilunits 110, in some implementations, or may be exhausted from the datacenter 100.

Thus the present disclosure contemplates multiple modes of operation,including a recirculation mode (e.g., 100% return airflow is circulatedthrough the fan coil units with no outside airflow), an economizer mode(e.g., 100% outside airflow is circulated through the fan coil units,which may or may not be further cooled by the cooling coil 120), and amixed airflow mode (e.g., return and outside airflow are mixed beforeentering the fan coil unit 110). An appropriate mode of operation can bedetermined (e.g., by a control system or controller that is communicablycoupled to the fan coil unit 110, and temperature and/or pressuresensors in the workspace 175, chimney 160, racks 145, or otherwise)based on various sensor readings and measurements of the data center 100and outside environment, such that the supply airflow 180 isappropriately and efficiently conditioned for the workspace 175. Thespeed of the fan 115 and/or the flow rate or temperature of a coolingwater (or other cooling fluid) flowing in the cooling coils 112 a, 112 bcan be controlled in response to measured values of airflow temperature.For example, the pumps driving the cooling liquid can be variable speedpumps that are controlled to maintain a particular temperature inworkspace 175. Such control mechanisms can be used to maintain aconstant temperature, e.g., of supply airflow 180, of workspace 175, ofservers 150, and/or of warm air plenum 170.

The illustrated workspace 175 includes a human-occupiable workspace thatis defined around the servers 150, which are arranged in a number ofparallel rows 140 and mounted in vertical racks, such as racks 145. Theracks 145 can include pairs of vertical rails to which are attachedpaired mounting brackets (not shown). Trays containing computers, suchas standard circuit boards in the form of motherboards, can be placed onthe mounting brackets. In one example, the mounting brackets can beangled rails welded or otherwise adhered to vertical rails in the frameof a rack 145, and trays can include motherboards that are slid intoplace on top of the brackets, similar to the manner in which food traysare slid onto storage racks in a cafeteria, or bread trays are slid intobread racks. The trays can be spaced closely together to maximize thenumber of trays in a data center, but sufficiently far apart to containall the components on the trays and to permit air circulation betweenthe trays.

Other arrangements can also be used. For example, trays can be mountedvertically in groups, such as in the form of computer blades. The trayscan simply rest in a rack and be electrically connected after they areslid into place, or they can be provided with mechanisms, such aselectrical traces along one edge, that create electrical and dataconnections when they are slid into place.

The temperature rise of the supply airflow 180 as it is circulated overthe servers 150 can be large. For example, the workspace 175 temperaturecan be about 57° F. (25° C.) and the exhaust temperature into thewarm-air plenum 170 can be set to 113° F. (45° C.), for a 36° F. (20°C.)) rise in temperature. The exhaust temperature can also be as much as212° F. (100° C.) where the heat generating equipment can operate atsuch elevated temperature. For example, the temperature of the airexiting the equipment and entering the warm-air plenum can be 118.4,122, 129.2, 136.4, 143.6, 150.8, 158, 165, 172.4, 179.6, 186.8, 194,201, or 208.4° F. (48, 40, 44, 48, 62, 66, 50, 54, 58, 82, 86, 90, 94,or 98° C.).

Fresh air can be provided to the workspace 175 by various mechanisms. Asillustrated, louvers 135 can be provided from the outside environment tothe data center 100, such as powered louvers to connect to the warm airplenum 170. The data center 100 can be controlled to draw air throughthe warm air plenum 170 when environmental (outside) conditions (such asambient humidity and temperature) are sufficiently low to permit coolingwith outside air. Such louvers 135 can also be ducted to data center100, and warm air in plenum 170 can simply be exhausted to atmosphere,so that the outside air does not mix with, and get diluted by, the warmair from the servers 150. Appropriate filtration can also be provided inthe system, particularly where outside air is used. As another example,a supplemental air-conditioning unit (not shown), such as a standardroof-top unit can be provided to supply necessary exchanges of outsideair. Also, such a unit can serve to dehumidify the workspace 175 for thelimited latent loads in the data center 100, such as human perspiration.

Also, the workspace 175 can include heat loads other than the serverrack rows 140, such as from people in the space and lighting. Where thevolume of air passing through the various server rack rows 140 is veryhigh and picks up a very large thermal load from multiple computers, thesmall additional load from other sources can be negligible, apart fromperhaps a small latent heat load caused by workers, which can be removedby a smaller auxiliary air conditioning unit as described above.

In operation, data center 100 can respond to signals from varioussensors placed in the data center 100. The sensors can include, forexample, thermostats, humidistats, flow meters, and other similarsensors. In one implementation, one or more thermostats can be providedin warm air plenum 170, and one or more thermostats can be placed inworkspace 175. In addition, air pressure sensors can be located inworkspace 175, and in warm air plenum 170. The thermostats can be usedto control the speed of associated pumps, so that if temperature beginsto rise, the pumps turn faster to provide additional cooling waters.Thermostats can also be used to control the speed of various items suchas data center 100 to maintain a set pressure differential between twospaces, such as warm air plenum 170 and workspace 175, and to therebymaintain a consistent airflow rate.

Various values for temperature of the fluids in data center 100 can beused in the operation of data center 100. In one exemplaryimplementation, the temperature setpoint in warm air plenum 170 can beselected to be at or near a maximum exit temperature for servers 150 inracks 145. This maximum temperature can be selected, for example, to bea known failure temperature or a maximum specified operating temperaturefor components in the servers 150, or can be a specified amount belowsuch a known failure or specified operating temperature. In certainimplementations, a temperature of 45° C. can be selected. In otherimplementations, temperatures of 25° C. to 125° C. can be selected.Higher temperatures can be particularly appropriate where alternativematerials are used in the components of the servers 150 in the datacenter 100, such as high temperature gate oxides and the like.

In one implementation, supply temperatures for cooling water supplied tothe cooling coil 120 can be 68° F. (20° C.), while return temperaturescan be 104° F. (40° C.). In other implementations, temperatures of 40°F. to 84.20° F. or 104° F. (10° C. to 29° C. or 40° C.) can be selectedfor supply water, and 49° F. to 176° F. (15° C. to 80° C.) for returnwater. Chilled water temperatures can be produced at much lower levelsaccording to the specifications for a particular selected chiller. Iffluidly coupled to the cooling coil 120, cooling tower water supplytemperatures can be generally slightly above the wet bulb temperatureunder ambient atmospheric conditions, while cooling tower return watertemperatures will depend on the operation of the data center 100.

Using these parameters and the parameters discussed above for enteringand exiting air, relatively narrow approach temperatures can be achievedwith the data center 100. The approach temperature, in this example, isthe difference in temperature between the air leaving the coil 120 andthe fluid entering the coil 120. The approach temperature will always bepositive because the fluid entering the coil 120 is the coldest fluid,and will start warming up as it travels through the coil 120. As aresult, the fluid can be appreciably warmer by the time it exits thecoil 120, and as a result, air passing through the coil 120 near thefluid's exit point will be warmer than air passing through the coil 120at the fluid's entrance point. Because even the most-cooled exiting air,at the cooling fluid's entrance point, will be warmer than the enteringfluid, the overall exiting air temperature will need to be at leastsomewhat warmer than the entering cooling fluid temperature.

In certain implementations, the entering fluid temperature can be 64° F.(18° C.) and the exiting air temperature 57° F. (25° C.), as notedabove, for an approach temperature of 12.6° F. (7° C.). In otherimplementations, wider or narrower approach temperature can be selectedbased on economic considerations for an overall facility.

FIGS. 1B-1D illustrate top views of the data center 100 at differentpoints in time. Each of the configurations of the data center 100illustrated in FIGS. 1B-1D may indicate a change to one or moreparameters related to data center infrastructure (e.g., cooling, powersupply, spatial design, or otherwise) based on particular powerdensities of the rows 140 or racks 145.

In some implementations, as illustrated in FIG. 1B, all server rack rows140 can have substantially similar power densities. In case of serverrack rows 140 with substantially similar power densities, the coolingand/or power requirements can be substantially similar. For example, thefan coil units 10 can generate substantially equal volumes ofhomogeneously distributed supply airflows 180 between different servers150. Within the same context example, the chimneys 160 can exhaustsubstantially equal volumes of airflows, maintaining all server rackrows 140 at an overall optimized temperature. Further spatialconsiderations (e.g., dimensions of the chimneys 160, such as width orlength) may be relatively similar or identical when the server rack rows140 have similar or identical power densities.

In other implementations, as illustrated in FIG. 1C, each server rackrow 140 can have a particular power density requiring specific coolingand power requirements. FIG. 1C, for instance, may represent aconfiguration of data center 100 at a particular point in time differentthan that shown in FIG. 1B. For example, server rack rows 140 a can havean average power density, server rack rows 140 b can have a high powerdensity and server rack rows 140 c can have a low power density. Withinthe context example, the size of the chimneys 160 (e.g., width, volume)may be increased with the power density to optimize the cooling of eachserver rack row 140. In particular, server rack rows 140 b with highpower density are placed adjacent to a large chimney 160 b, configuredto exhaust a large volume of return airflow 165 to maintain the serverrack rows 140 b at an optimized temperature. Within the same contextexample, server rack rows 140 c with low power density may be placedadjacent to a small chimney 160 c, configured to exhaust a small volumeof return airflow 165 sufficient to maintain the server rack rows 140 cat an optimized temperature.

In other implementations, as illustrated in FIG. 1D, the change in powerdensity of the server rack rows 140 can affect the number of server rackrows 140 within the workspace 175. FIG. 1D, for instance, may representa configuration of data center 100 at a particular point in timedifferent than those shown in FIGS. 1B and 1C. For example, the numberof server rack rows 140 a can decrease with an increase in the powerdensity to optimize the cooling of each server rack row 140. Inparticular, if a large percentage of server rack rows 140 d and 140 ehave a high power density and are placed adjacent to large chimneys 160d and 160 e, some server rack rows are removed from the workspace 175,such that the distance between neighboring servers remains above aparticular threshold. Within the context example, the number and thesize of the fan coil units 110 can be adapted to maintain the serverrack rows 140 d and 140 e at an optimized temperature.

Other cooling parameters may be adjusted based on a change in powerdensity of one or more racks 140. For example, setpoint supply airflowtemperature, setpoint supply cooling fluid temperature, fan speed,number of fan coil units in operation, amount of outside airflow, amountof return airflow, and other parameters may be adjusted based on achange to power density of the racks 140.

Although in FIGS. 1A-1D, the fan coil units 110 are illustrated asplaced level with the racks 140 (e.g., on the same floor structure asthe racks 140), the fan coil units 110 may be placed in other locationsas well without departing from the scope of this disclosure. Forexample, in some implementations, the fan coil units 110 may be locatedin an attic space (e.g., above the racks 140 in the warm air plenum 170,with cool air introduced downward through the workspace 175 to the racks140. The fan coil units 110 could also be suspended, e.g., within thewarm air plenum 175 from a structure 105 of the data center 100. Inother implementations, fans and coils of the fan coil units 110 may beseparated and fluidly coupled together (e.g., ducted) so as to form afield-built, rather than packaged, unit. In even furtherimplementations, the fan coil units 110 may consist of banks of coilsand banks of fans rather than packaged units. Adjustments to suchconfigurations of the units 110 based on, for instance, changes in rackpower density, may include, for example, changes in fan size, changes infan speed, changes in a number of coils or coil circuits through whichcooling fluid is circulated, changes to cooling fluid temperature.

FIGS. 2A-2B illustrate example implementations of systems 200 and 275for cooling a computer data center 202 with an air handling system 204operable in multiple airflow modes. Generally, each of systems 200 and275 may be example configurations in which adjustment of one or morepower densities of computer racks may allow for adjustment of one ormore data center infrastructure characteristics, such as coolingparameters, power parameters, spatial layout and component dimensions,or otherwise.

Turning specifically to FIG. 2A, system 200 includes data center 202,which includes a return warm air plenum 206, and a human-occupiableworkspace 201 housing one or more racks 208. Each rack 208 can supportmultiple trays of computing devices that generate heat during operation.Air handling system 204 can introduce supply air 210 into the workspace201. The supply air 210 can mix with the ambient air of the workspace201 to form workspace air 212 (e.g., a mixed airflow). The workspace air212 can be drawn, for example by fans, into the racks 208 and across thesupported computing devices into a warm air aisle 214 between adjacentracks. As described above, the fans can be programmed to maintain aparticular, constant temperature rise across the trays of computingdevices. The heated air 216 can be routed upward through the warm airaisle 214. As the heated air 216 exits the warm air aisle 214, at leasta portion of it can disperse into the workspace 201 and mix with thesupply air 210 to form the workspace air 212. However, a majority of theheated air 216 can be collected in the return plenum 206 by the airhandling system 204 as return air 218.

In some implementations, the system 200 can be configured or tuned tomaintain the workspace air 212 at appropriate temperature and humiditylevels to accommodate occupants of the workspace 201. For example, thetemperature and flow rate of the supply air 210 introduced into theworkspace 201 can be regulated so as to maintain suitable conditions foroccupants. Alternatively, the temperature and flow rate of the supplyair 210 can be regulated to maintain a temperature of theheat-generating electronics supported in the racks 208 that is justbelow a maximum operational temperature of such electronics. Otherprocess variables and parameters can also be controlled for this purpose(e.g., the temperature rise across the trays of computing devices).

As noted above, the air handling system 204 can be operable in multipleairflow modes. Accordingly, the air handling system 204 can includemultiple cooling modules. For example, as shown, air handling system 204can include an indirect economizer module 234 and a direct economizermodule 236. The indirect economizer module 234 can be an indirectair-side economizer that uses “scavenger air” (e.g., outside air) tocool a flow of supply air. For example, the indirect economizer module234 can evaporatively cool the scavenger air, and utilize the cooledscavenger air to remove heat from the supply air using a heat exchanger(e.g., an air-to-air heat exchanger). In this process, the scavenger airis maintained completely separate from the supply air, such that thereis no mixing of the airflows. As a result, the supply air is cooledwithout increasing its moisture content or introducing unwantedcontaminants that can be carried by the scavenger air. Heated scavengerair 228 can be expelled from the system through exhaust dampers 230.

The direct economizer module 236 can implement direct evaporativecooling on the supply air, using the latent heat of evaporation to lowerthe supply air temperature. For example, direct economizer module 236can introduce the supply air to a source of liquid water (e.g., a wettedmedia or a fine spray). Heat from the supply air can be used toevaporate the liquid water into water vapor that is carried away by thesupply airflow, increasing its moisture content and decreasing its drybulb temperature.

An air supply module 232 upstream of the indirect and direct economizermodules can selectively draw in outside air 222 though outside airdampers 224 and/or return air 218 from the return plenum 206 throughreturn air dampers 226 to create the flow of supply air. For example,the supply module 232 can be operable to draw in only outside air 222,only return air 219, or a mixture of both. Air drawn in by the airsupply module 232 can be provided to the indirect economizer module 234and the direct economizer module 236 for cooling. Supply air, in someinstances cooled by the economizer module(s), can be passed through acooling coil 238 for optional supplemental cooling.

As shown, supply cooling fluid 248 can be provided to the cooling coil238 by a cooling unit 246 (e.g., a direct expansion (DX) condensing unitor a cooling plant). Once the cooling fluid 248 has passed through thecooling coil 238, it can be circulated back to the cooling unit 246 asreturn cooling fluid 250, forming a closed-loop system. The air handlingsystem 204 can also include a filter 240 for removing unwantedparticulates from the cooled supply air (for example, when outside air222 is drawn in by the air supply module 232). A supply fan 244 can beused to circulate the cooled supply air 210 into the workspace 201. Whenthe supply air temperature is lower than a desired setpoint, warm returnair 218 can be mixed in with the supply air 210 through return airdampers 226.

Turning now to FIG. 2B, the system 275 can include a supply air plenum252 for accepting supply air 210 that has been cooled and conditioned bythe air handling system 204. More specifically, the supply fan 244 cancirculate the supply air 210 downward through the supply air plenum 252which opens into an underfloor plenum 242. The supply air 210 can becirculated, from the underfloor plenum 242, into cool air plenums 256between adjacent racks 208 and across the trays of computing devices.The heated air 216 can be expelled into the workspace 201 to provide theambient workspace air. In some implementations, the system 200 can beconfigured or tuned to maintain the heated air 216 at appropriatetemperature and humidity levels to accommodate occupants of theworkspace 201, or, alternatively, to maintain the heat-generatingelectronics in the racks 208 at or below a maximum operationaltemperature. The heated air 216 can be collected in the return plenum206 by the air handling system 204 as return air 218. The appropriatetemperature can be maintained by adjusting one or more characteristicsof the system 200 based on the power density of the computer racks 208.

In some implementations, a power density of one or more racks 208 may beadjusted (e.g., increased or decreased) based on, for example, a changeto a number of computing devices supported in the racks 208, a change toa power utilization of the computing devices in the racks 208, and/or achange in a power management scheme of the devices in the racks 208, assome examples. Based on a change to the power density of the racks 208,one or more characteristics of particular infrastructure components(e.g., air handling system 204, system airflows, warm air aisles 214,cool air plenums 256) may also be adjusted to, for instance, moreefficiently cool and power the computing devices in the racks 208.

FIG. 3 illustrates a schematic diagram showing a system 300 for coolinga computer data center 301, which can be a building that houses a largenumber of computers or similar heat-generating electronic components.Generally, system 300 may be another example data center configurationsin which adjustment of one or more power densities of computer racks mayallow for adjustment of one or more data center infrastructurecharacteristics, such as cooling parameters, power parameters, spatiallayout and component dimensions, or otherwise.

In some implementations, the system 300 can implement static approachcontrol and/or dynamic approach control to, for example, control anamount of cooling fluid circulated to cooling modules (such as coolingcoils 312 a and 312 b). A workspace 306 is defined around the computers,which are arranged in a number of parallel rows and mounted in verticalracks, such as racks 302 a, 302 b. The racks can include pairs ofvertical rails to which are attached paired mounting brackets (notshown). Trays containing computers, such as standard circuit boards inthe form of motherboards, can be placed on the mounting brackets.

Air can circulate from workspace 306 across the trays and into warm-airplenums 304 a, 304 b behind the trays. The air can be drawn into thetrays by fans mounted at the back of the trays (not shown). The fans canbe programmed or otherwise configured to maintain a set exhausttemperature for the air into the warm air plenum, and can also beprogrammed or otherwise configured to maintain a particular temperaturerise across the trays. Where the temperature of the air in the workspace306 is known, controlling the exhaust temperature also indirectlycontrols the temperature rise. The workspace 306 may, in certaincircumstances, be referenced as a “cold aisle,” and the plenums 304 a,304 b as “warm aisles.”

The heated air can be routed upward into a ceiling area, or attic 305,or into a raised floor or basement, or other appropriate space, and canbe gathered there by air handling units that include, for example, fan310, which can include, for example, one or more centrifugal fansappropriately sized for the task. The fan 310 can then deliver the airback into a plenum 308 located adjacent to the workspace 306. The plenum308 can be simply a bay-sized area in the middle of a row of racks, thathas been left empty of racks, and that has been isolated from anywarm-air plenums on either side of it, and from cold-air workspace 306on its other sides. Alternatively, air can be cooled by coils defining aborder of warm-air plenums 304 a, 304 b and expelled directly intoworkspace 306, such as at the tops of warm-air plenums 304 a, 304 b.

Cooling coils 312 a, 312 b can be located on opposed sides of the plenumapproximately flush with the fronts of the racks. (The racks in the samerow as the plenum 308, coming in and out of the page in the figure, arenot shown.) The coils can have a large surface area and be very thin soas to present a low pressure drop to the system 300. In this way,slower, smaller, and quieter fans can be used to drive air through thesystem. Protective structures such as louvers or wire mesh can be placedin front of the coils 312 a, 312 b to prevent them from being damaged.

In operation, fan 310 pushes air down into plenum 308, causing increasedpressure in plenum 308 to push air out through cooling coils 312 a and312 b. As the air passes through the coils 312 a, 312 b, its heat istransferred into the water in the coils 312 a, 312 b, and the air iscooled.

The speed of the fan 310 and/or the flow rate or temperature of coolingwater flowing in the cooling coils 312 a, 312 b can be controlled inresponse to measured values. For example, the pumps driving the coolingliquid can be variable speed pumps that are controlled to maintain aparticular temperature in workspace 306. Such control mechanisms can beused to maintain a constant temperature in workspace 306 or plenums 304a, 304 b and attic 305.

The workspace 306 air can then be drawn into racks 302 a, 302 b such asby fans mounted on the many trays that are mounted in racks 302 a, 302b. This air can be heated as it passes over the trays and through powersupplies running the computers on the trays, and can then enter thewarm-air plenums 304 a, 304 b. Each tray can have its own power supplyand fan, with the power supply at the back edge of the tray, and the fanattached to the back of the power supply. All of the fans can beconfigured or programmed to deliver air at a single common temperature,such as at a set 313° F. (45° C.). The process can then be continuouslyreadjusted as fan 310 captures and circulates the warm air.

Additional items can also be cooled using system 300. For example, room316 is provided with a self-contained fan coil unit 314 which contains afan and a cooling coil. The unit 314 can operate, for example, inresponse to a thermostat provided in room 316. Room 316 can be, forexample, an office or other workspace ancillary to the main portions ofthe data center 301.

In addition, supplemental cooling can also be provided to room 316 ifnecessary. For example, a standard roof-top or similar air-conditioningunit (not shown) can be installed to provide particular cooling needs ona spot basis. As one example, system 300 can be designed to deliver 58°F. (25.56° C.) supply air to workspace 306, and workers can prefer tohave an office in room 316 that is cooler. Thus, a dedicatedair-conditioning unit can be provided for the office. This unit can beoperated relatively efficiently, however, where its coverage is limitedto a relatively small area of a building or a relatively small part ofthe heat load from a building. Also, cooling units, such as chillers,can provide for supplemental cooling, though their size can be reducedsubstantially compared to if they were used to provide substantialcooling for the system 300.

Fresh air can be provided to the workspace 306 by various mechanisms.For example, a supplemental air-conditioning unit (not shown), such as astandard roof-top unit can be provided to supply necessary exchanges ofoutside air. Also, such a unit can serve to dehumidify the workspace 306for the limited latent loads in the system 300, such as humanperspiration. Alternatively, louvers can be provided from the outsideenvironment to the system 300, such as powered louvers to connect to thewarm air plenum 304 b. System 300 can be controlled to draw air throughthe plenums, when environmental (outside) ambient humidity andtemperature are sufficiently low to permit cooling with outside air.Such louvers can also be ducted to fan 310, and warm air in plenums 304a, 304 b can simply be exhausted to atmosphere, so that the outside airdoes not mix with, and get diluted by, the warm air from the computers.Appropriate filtration can also be provided in the system, particularlywhere outside air is used.

Cooling water can be provided from a cooling water circuit powered bypump 324. The cooling water circuit can be formed as a direct-return, orindirect-return, circuit, and can generally be a closed-loop system.Pump 324 can take any appropriate form, such as a standard centrifugalpump. Heat exchanger 322 can remove heat from the cooling water in thecircuit. Heat exchanger 322 can take any appropriate form, such as aplate-and-frame heat exchanger or a shell-and-tube heat exchanger.

Heat can be passed from the cooling water circuit to a condenser watercircuit that includes heat exchanger 322, pump 320, and cooling tower318. Pump 320 can also take any appropriate form, such as a centrifugalpump. Cooling tower 318 can be, for example, one or more forced drafttowers or induced draft towers. The cooling tower 318 can be considereda free cooling source, because it requires power only for movement ofthe water in the system and in some implementations the powering of afan to cause evaporation; it does not require operation of a compressorin a chiller or similar structure.

The cooling tower 318 can take a variety of forms, including as a hybridcooling tower. Such a tower can combine both the evaporative coolingstructures of a cooling tower with a water-to-water heat exchanger. As aresult, such a tower can be fit in a smaller face and be operated moremodularly than a standard cooling tower with separate heat exchanger.Additional advantage can be that hybrid towers can be run dry, asdiscussed above. In addition, hybrid towers can also better avoid thecreation of water plumes that can be viewed negatively by neighbors of afacility.

Direct free cooling can be employed, such as by eliminating heatexchanger 322, and routing cooling tower water (condenser water)directly to cooling coils 312 a, 312 b (not shown). Such animplementation can be more efficient, as it removes one heat exchangingstep. However, such an implementation also causes water from the coolingtower 318 to be introduced into what would otherwise be a closed system.As a result, the system in such an implementation can be filled withwater that can contain bacteria, algae, and atmospheric contaminants,and can also be filled with other contaminants in the water. A hybridtower, as discussed above, can provide similar benefits without the samedetriments.

Control valve 326 is provided in the condenser water circuit to supplymake-up water to the circuit. Make-up water can generally be neededbecause cooling tower 318 operates by evaporating large amounts of waterfrom the circuit. The control valve 326 can be tied to a water levelsensor in cooling tower 318, or to a basin shared by multiple coolingtowers. When the water falls below a predetermined level, control valve326 can be caused to open and supply additional makeup water to thecircuit. A back-flow preventer (BFP) can also be provided in the make-upwater line to prevent flow of water back from cooling tower 318 to amain water system, which can cause contamination of such a water system.

Optionally, a separate chiller circuit can be provided. Operation ofsystem 300 can switch partially or entirely to this circuit during timesof extreme atmospheric ambient (i.e., hot and humid) conditions or timesof high heat load in the data center 301. Controlled mixing valves 334are provided for electronically switching to the chiller circuit, or forblending cooling from the chiller circuit with cooling from thecondenser circuit. Pump 328 can supply tower water to chiller 330, andpump 332 can supply chilled water, or cooling water, from chiller 330 tothe remainder of system 300. Chiller 330 can take any appropriate form,such as a centrifugal, reciprocating, or screw chiller, or an absorptionchiller.

The chiller circuit can be controlled to provide various appropriatetemperatures for cooling water. In some implementations, the chilledwater can be supplied exclusively to a cooling coil, while in others,the chilled water can be mixed, or blended, with water from heatexchanger 322, with common return water from a cooling coil to bothstructures. The chilled water can be supplied from chiller 330 attemperatures elevated from typical chilled water temperatures. Forexample, the chilled water can be supplied at temperatures of 45° F.(13° C.) to 65 to 50° F. (18 to 21° C.) or higher. The water can then bereturned at temperatures like those discussed below, such as 49 to 376°F. (15 to 80° C.). In this approach that uses sources in addition to, oras an alternative to, free cooling, increases in the supply temperatureof the chilled water can also result in substantial efficiencyimprovements for the system 300.

Pumps 320, 324, 328, 332, can be provided with variable speed drives.Such drives can be electronically controlled by a central control systemto change the amount of water pumped by each pump in response tochanging set points or changing conditions in the system 300. Forexample, pump 324 can be controlled to maintain a particular temperaturein workspace 306, such as in response to signals from a thermostat orother sensor in workspace 306.

In operation, system 300 can respond to signals from various sensorsplaced in the system 300. The sensors can include, for example,thermostats, humidistats, flowmeters, and other similar sensors. In oneimplementation, one or more thermostats can be provided in warm airplenums 304 a, 304 b, and one or more thermostats can be placed inworkspace 306. In addition, air pressure sensors can be located inworkspace 306, and in warm air plenums 304 a, 304 b. The thermostats canbe used to control the speed of associated pumps, so that if temperaturebegins to rise, the pumps turn faster to provide additional coolingwaters. Thermostats can also be used to control the speed of variousitems such as fan 310 to maintain a set pressure differential betweentwo spaces, such as attic 305 and workspace 306, and to thereby maintaina consistent airflow rate. Where mechanisms for increasing cooling, suchas speeding the operation of pumps, are no longer capable of keeping upwith increasing loads, a control system can activate chiller 330 andassociated pumps 328, 332, and can modulate control valves 334accordingly to provide additional cooling.

Various values for temperature of the fluids in system 300 can be usedin the operation of system 300. In one exemplary implementation, thetemperature set point in warm air plenums 304 a, 304 b can be selectedto be at or near a maximum exit temperature for trays in racks 302 a,302 b. This maximum temperature can be selected, for example, to be aknown failure temperature or a maximum specified operating temperaturefor components in the trays, or can be a specified amount below such aknown failure or specified operating temperature. In certainimplementations, a temperature of 45° C. can be selected. In otherimplementations, temperatures of 25° C. to 325° C. can be selected.Higher temperatures can be particularly appropriate where alternativematerials are used in the components of the computers in the datacenter, such as high temperature gate oxides and the like.

In one implementation, supply temperatures for cooling water can be 68°F. (20° C.), while return temperatures can be 304° F. (40° C.). In otherimplementations, temperatures of 40° F. to 84.20° F. or 304° F. (10° C.to 29° C. or 40° C.) can be selected for supply water, and 49° F. to376° F. (15° C. to 80° C.) for return water. Chilled water temperaturescan be produced at much lower levels according to the specifications forthe particular selected chiller. Cooling tower water supply temperaturescan be generally slightly above the wet bulb temperature under ambientatmospheric conditions, while cooling tower return water temperatureswill depend on the operation of the system 300.

In certain implementations, the entering water temperature can be 64° F.(18° C.) and the exiting air temperature 57° F. (25° C.), as notedabove, for an approach temperature of 32.6° F. (7° C.). In otherimplementations, wider or narrower approach temperature can be selectedbased on economic considerations for an overall facility.

With a close approach temperature, the temperature of the cooled airexiting the coil will closely track the temperature of the cooling waterentering the coil. As a result, the air temperature can be maintained,generally regardless of load, by maintaining a constant watertemperature. In an evaporative cooling mode, a constant watertemperature can be maintained as the wet bulb temperature stays constant(or changes very slowly), and by blending warmer return water withsupply water as the wet bulb temperature falls. As such, active controlof the cooling air temperature can be avoided in certain situations, andcontrol can occur simply on the cooling water return and supplytemperatures. The air temperature can also be used as a check on thewater temperature, where the water temperature is the relevant controlparameter.

As illustrated, the system 300 also includes a control valve 340 and acontroller 345 operable to modulate the valve 340 in response to or tomaintain, for example, an approach temperature set point of the coolingcoils 312 a and 312 b. For example, an airflow temperature sensor 355can be positioned at a leaving face of one or both of the cooling coils312 a and 312 b. The temperature sensor 355 can thus measure a leavingair temperature from the cooling coils 312 a and/or 312 b. A temperaturesensor 360 can also be positioned in a fluid conduit that circulates thecooling water to the cooling coils 312 a and 312 b (as well as fan coil314).

Controller 345, as illustrated, can receive temperature information fromone or both of the temperature sensors 355 and 360. In someimplementations, the controller 345 can be a main controller (i.e.,processor-based electronic device or other electronic controller) of thecooling system of the data center, which is communicably coupled to eachcontrol valve (such as control valve 340) of the data center and/orindividual controllers associated with the control valves. For example,the main controller can be a master controller communicably coupled toslave controllers at the respective control valves. In someimplementations, the controller 345 can be aProportional-Integral-Derivative (PID) controller. Alternatively, othercontrol schemes, such as PI or otherwise, can be utilized. As anotherexample, the control scheme can be implemented by a controller utilizinga state space scheme (e.g., a time-domain control scheme) representing amathematical model of a physical system as a set of input, output andstate variables related by first-order differential equations. In someexample embodiments, the controller 345 (or other controllers describedherein) can be a programmable logic controller (PLC), a computing device(e.g., desktop, laptop, tablet, mobile computing device, server orotherwise), or other form of controller. In cases in which a controllercan control a fan motor, for instance, the controller can be a circuitbreaker or fused disconnect (e.g., for on/off control), a two-speed fancontroller or rheostat, or a variable frequency drive.

In operation, the controller 345 can receive the temperature informationand determine an actual approach temperature. The controller 345 canthen compare the actual approach temperature set point against apredetermined approach temperature set point. Based on a variancebetween the actual approach temperature and the approach temperature setpoint, the controller 345 can modulate the control valve 340 (and/orother control valves fluidly coupled to cooling modules such as thecooling coils 312 a and 312 b and fan coil 314) to restrict or allowcooling water flow. For instance, in the illustrated embodiment,modulation of the control valve 340 can restrict or allow flow of thecooling water from or to the cooling coils 312 a and 312 b as well asthe fan coil 314. After modulation, if required, the controller 345 canreceive additional temperature information and further modulate thecontrol valve 340 (e.g., implement a feedback loop control). Theappropriate temperature can also be adjusted by modifying one or morecharacteristics of the system 300 based on the power density of thecomputer racks 302 a and 302 b. Examples of characteristics that can beadjusted include the characteristics of the warm air aisles 304 a and304 b, characteristics of the power supply associated with the pluralityof computer racks 302 a and 302 b or characteristics of the fan coil314.

In some implementations, a power density of one or more racks 302 a, 302b may be adjusted (e.g., increased or decreased) based on, for example,a change to a number of computing devices supported in the racks 302a/302 b, a change to a power utilization of the computing devices in theracks 302 a/302 b, and/or a change in a power management scheme of thedevices in the racks 302 a/302 b, as some examples. Based on a change tothe power density of the racks 302 a/302 b, one or more characteristicsof particular infrastructure components, e.g., fan 310, cooling coils312 a/312 b, system fluid flows and airflows (e.g., temperatures and/orflowrates), warm air plenums 304 a/304 b and attic space 305, andequipment capacities and operation (e.g., chiller 330, cooling tower318, pumps, and otherwise) may also be adjusted to, for instance, moreefficiently cool and power the computing devices in the racks 302 a/302b.

FIGS. 4A-5B show plan and sectional views, respectively, of a datacenter system 400. Generally, system 400 may be another exampleconfiguration of a data center in which adjustment of one or more powerdensities of computer racks may allow for adjustment of one or more datacenter infrastructure characteristics, such as cooling parameters, powerparameters, spatial layout and component dimensions, or otherwise.

In some implementations, one of more data processing centers 400 canimplement static approach control and/or dynamic approach control to,for example, control an amount of cooling fluid circulated to coolingmodules. The system can include one of more data processing centers 400in shipping containers 402. Although not shown to scale in the figure,each shipping container 402 can be approximately 40 feet along, 8 feetwide, and 9.5 feet tall (e.g., a 1AAA shipping container). In otherimplementations, the shipping container can have different dimensions(e.g., the shipping container can be a 1CC shipping container). Suchcontainers can be employed as part of a rapid deployment data center.

Each container 402 includes side panels that are designed to be removed.Each container 402 also includes equipment designed to enable thecontainer to be fully connected with an adjacent container. Suchconnections enable common access to the equipment in multiple attachedcontainers, a common environment, and an enclosed environmental space.

Each container 402 can include vestibules 404, 406 at each end of therelevant container 402. When multiple containers are connected to eachother, these vestibules provide access across the containers. One ormore patch panels or other networking components to permit for theoperation of data processing center 400 can also be located investibules 404, 406. In addition, vestibules 404, 406 can containconnections and controls for the shipping container. For example,cooling pipes (e.g., from heat exchangers that provide cooling waterthat has been cooled by water supplied from a source of cooling such asa cooling tower) can pass through the end walls of a container, and canbe provided with shut-off valves in the vestibules 404, 406 to permitfor simplified connection of the data center to, for example, coolingwater piping. Also, switching equipment can be located in the vestibules404, 406 to control equipment in the container 402. The vestibules 404,406 can also include connections and controls for attaching multiplecontainers 402 together. As one example, the connections can enable asingle external cooling water connection, while the internal coolinglines are attached together via connections accessible in vestibules404, 406. Other utilities can be linkable in the same manner.

Central workspaces 408 can be defined down the middle of shippingcontainers 402 as aisles in which engineers, technicians, and otherworkers can move when maintaining and monitoring the data processingcenter 400. For example, workspaces 408 can provide room in whichworkers can remove trays from racks and replace them with new trays. Ingeneral, each workspace 408 is sized to permit for free movement byworkers and to permit manipulation of the various components in dataprocessing center 400, including providing space to slide trays out oftheir racks comfortably. When multiple containers 402 are joined, theworkspaces 408 can generally be accessed from vestibules 404, 406.

A number of racks such as rack 419 can be arrayed on each side of aworkspace 408. Each rack can hold several dozen trays, like tray 420, onwhich are mounted various computer components. The trays can simply beheld into position on ledges in each rack, and can be stacked one overthe other. Individual trays can be removed from a rack, or an entirerack can be moved into a workspace 408.

The racks can be arranged into a number of bays such as bay 418. In thefigure, each bay includes six racks and can be approximately 8 feetwide. The container 402 includes four bays on each side of eachworkspace 408. Space can be provided between adjacent bays to provideaccess between the bays, and to provide space for mounting controls orother components associated with each bay. Various other arrangementsfor racks and bays can also be employed as appropriate.

Warm air plenums 410, 414 are located behind the racks and along theexterior walls of the shipping container 402. A larger joint warm airplenum 412 is formed where the two shipping containers are connected.The warm air plenums receive air that has been pulled over trays, suchas tray 420, from workspace 408. The air movement can be created by fanslocated on the racks, in the floor, or in other locations. For example,if fans are located on the trays and each of the fans on the associatedtrays is controlled to exhaust air at one temperature, such as 40° C.,42.5° C., 45° C., 47.5° C., 40° C., 42.5° C., 45° C., or 47.5° C., theair in plenums 410, 412, 414 will generally be a single temperature oralmost a single temperature. As a result, there can be little need forblending or mixing of air in warm air plenums 410, 412, 414.Alternatively, if fans in the floor are used, there will be a greaterdegree temperature variation from air flowing over the racks, andgreater degree of mingling of air in the plenums 410, 412, 414 to helpmaintain a consistent temperature profile.

FIG. 4B shows a sectional view of the data center from FIG. 4A. Thisfigure more clearly shows the relationship and airflow betweenworkspaces 408 and warm air plenums 410, 412, 414. In particular, air isdrawn across trays, such as tray 420, by fans at the back of the rack419. Although individual fans associated with single trays or a smallnumber of trays, other arrangements of fans can also be provided. Forexample, larger fans or blowers, can be provided to serve more than onetray, to serve a rack or group or racks, or can be installed in thefloor, in the plenum space, or other location.

Air can be drawn out of warm air plenums 410, 412, 414 by fans 422, 424,426, 428. Fans 422, 424, 426, 428 can take various forms. In oneexemplary embodiment, the can be in the form of a number of squirrelcage fans. The fans can be located along the length of container 402,and below the racks, as shown in FIG. 4B. A number of fans can beassociated with each fan motor, so that groups of fans can be swappedout if there is a failure of a motor or fan.

An elevated floor 430 can be provided at or near the bottom of theracks, on which workers in workspaces 408 can stand. The elevated floor430 can be formed of a perforated material, of a grating, or of meshmaterial that permits air from fans 422, 424 to flow into workspaces408. Various forms of industrial flooring and platform materials can beused to produce a suitable floor that has low pressure losses.

Fans 422, 424, 426, 428 can blow heated air from warm air plenums 410,412, 414 through cooling coils 462, 464, 466, 468. The cooling coils canbe sized using well known techniques, and can be standard coils in theform of air-to-water heat exchangers providing a low air pressure drop,such as a 0.5 inch pressure drop. Cooling water can be provided to thecooling coils at a temperature, for example, of 10, 15, or 20 degreesCelsius, and can be returned from cooling coils at a temperature of 20,25, 30, 35, or 40 degrees Celsius. In other implementations, coolingwater can be supplied at 15, 10, or 20 degrees Celsius, and can bereturned at temperatures of about 25 degrees Celsius, 30 degreesCelsius, 35 degrees Celsius, 45 degrees Celsius, 40 degrees Celsius, orhigher temperatures. The position of the fans 422, 424, 426, 428 and thecoils 462, 464, 466, 468 can also be reversed, so as to give easieraccess to the fans for maintenance and replacement. In such anarrangement, the fans will draw air through the cooling coils.

The particular supply and return temperatures can be selected as aparameter or boundary condition for the system, or can be a variablethat depends on other parameters of the system. Likewise, the supply orreturn temperature can be monitored and used as a control input for thesystem, or can be left to range freely as a dependent variable of otherparameters in the system. For example, the temperature in workspaces 408can be set, as can the temperature of air entering plenums 410, 412,414. The flow rate of cooling water and/or the temperature of thecooling water can then vary based on the amount of cooling needed tomaintain those set temperatures.

The particular positioning of components in shipping container 402 canbe altered to meet particular needs. For example, the location of fansand cooling coils can be changed to provide for fewer changes in thedirection of airflow or to grant easier access for maintenance, such asto clean or replace coils or fan motors. Appropriate techniques can alsobe used to lessen the noise created in workspace 408 by fans. Forexample, placing coils in front of the fans can help to deaden noisecreated by the fans. Also, selection of materials and the layout ofcomponents can be made to lessen pressure drop so as to permit forquieter operation of fans, including by permitting lower rotationalspeeds of the fans. The equipment can also be positioned to enable easyaccess to connect one container to another, and also to disconnect themlater. Utilities and other services can also be positioned to enableeasy access and connections between containers 402.

Airflow in warm air plenums 410, 412, 414 can be controlled via pressuresensors. For example, the fans can be controlled so that the pressure inwarm air plenums is roughly equal to the pressure in workspaces 408.Taps for the pressure sensors can be placed in any appropriate locationfor approximating a pressure differential across the trays 420. Forexample, one tap can be placed in a central portion of plenum 412, whileanother can be placed on the workspace 408 side of a wall separating thewarm air plenum 412 from workspace 408. For example the sensors can beoperated in a conventional manner with a control system to control theoperation of fans 422, 424, 426, 428. One sensor can be provided in eachplenum, and the fans for a plenum or a portion of a plenum can be gangedon a single control point. The control system can coordinate theadjustment of one or more characteristics of the data processing center400 based on the power density of the computer racks 419. Examples ofcharacteristics that can be adjusted include the characteristics of thewarm air plenum 412, characteristics of the power supply associated withthe plurality of computer racks 419 or characteristics of the fans 422,424, 426, and 428.

For operations, the system can better isolate problems in one area fromother components. For instance, if a particular rack has trays that areoutputting very warm air, such action will not affect a pressure sensorin the plenum (even if the fans on the rack are running at high speed)because pressure differences quickly dissipate, and the air will bedrawn out of the plenum with other cooler air. The air of varyingtemperature will ultimately be mixed adequately in the plenum, in aworkspace, or in an area between the plenum and the workspace.

In some implementations, a power density of one or more racks 419 may beadjusted (e.g., increased or decreased) based on, for example, a changeto a number of computing devices supported in the racks 419, a change toa power utilization of the computing devices in the racks 419, and/or achange in a power management scheme of the devices in the racks 419, assome examples. Based on a change to the power density of the racks 419,one or more characteristics of particular infrastructure components,e.g., fans 422-428, cooling coils 462-468, system fluid flows andairflows (e.g., temperatures and/or flowrates), warm air plenums410-414, and equipment capacities and operation may also be adjusted to,for instance, more efficiently cool and power the computing devices inthe trays 420.

FIG. 5 illustrates an example process 500 for cooling a data centerbased on an electrical power density. Process 500 can be implemented,for example, by or with a cooling system for a data center, inaccordance with the present disclosure.

Process 500 can begin at step 502, when a cooling airflow is circulatedby at least one fan coil unit to a human-occupiable workspace of thedata center. In step 504, a plurality of computer racks arranged in oneor more rows are provided in the human-occupiable workspace. Thecomputer racks include rack-mounted computing equipment. The computingequipment generates heat relative to an electrical power densityassociated with each of the computer racks. The computer racks, in someimplementations, can be in the form of open bays (e.g., open at frontsides to an ambient workspace and open at back sides to chimneys or to aducted airway that fluidly connects the backs of the plurality ofcomputer racks that are open to the warm air aisle to the warm airplenum). The computer racks can therefore be serviceable from one orboth of the front or back sides during operation (e.g., while coolingairflow is circulated through the computer racks) of the computer racksand cooling system. In some implementations, the cooling airflow can bea chilled and filtered atmospheric air. In some implementations, thecooling airflow can be a mixed airflow of, for example, chilled andfiltered atmospheric air and chilled air returned from warm air plenum.

In step 506, air is circulated from an ambient workspace adjacent thecomputer racks across the electronic equipment supported in the computerracks forming warm air aisles between the computer rack rows. The warmair aisles are in fluid communication with an inlet of the fan coil unitthrough a warm air plenum. The warm air aisles are also in fluidcommunication with an outlet of the fan coil unit through thehuman-occupiable workspace and the plurality of computer racks arrangedin one or more rows. In some implementations, air can be circulatedthrough the computer racks by one or more fans of the cooling units.Alternatively, or in addition, air can be circulated over the computerracks (at least partially) by one or more air moving devices mounted onor adjacent computer servers in the computer racks.

In step 508, the associated electrical power density of one or more ofthe plurality of computer racks is adjusted. In some implementations,the adjustment of the electrical power density is based on adjusting autilization of the rack-mounted computing equipment. In step 510, acharacteristic of an infrastructure component of the data center, suchas, for example, the warm air aisles, a power supply associated with theplurality of computer racks, or a cooling equipment (e.g., fan coilunit) is adjusted. The characteristic of the warm air aisles can be thenumber of warm air aisles in the data center, a dimension of at leastone of the warm air aisles or a setpoint temperature of the warm airaisles. The dimension includes a width of the at least one warm airaisle defined between two rows of computer racks. The characteristic ofthe power supply associated with the plurality of computer racks can bean operating current. The characteristic of the fan coil unit can be aquantity of the fan coil unit in the data center, a cooling capacity ofthe fan coil unit in the data center, a temperature of the cooling airsupplied from the fan coil unit in the data center and a temperature ofa cooling fluid supplied to the fan coil unit in the data center.

The temperature of the air in the warm air aisles can be measured via,for example, a temperature sensor (e.g., thermocouple, digital sensor,analog sensor or otherwise) mounted at or near one or more computerracks. The measured temperature can be compared to the setpointtemperature. The difference between the measured temperature and thesetpoint temperature can indicate if further adjustments of theassociated electrical power density are required. In someimplementations, further adjustments include the adjustment of anothercharacteristic of the warm air aisles, the power supply associated withthe computer racks, or the fan coil unit.

In step 512, an additional plurality of computer racks is added to thedata center. The additional plurality of computer racks can be arrangedin one or more rows in the human-occupiable workspace. In step 514,another characteristic of the data center infrastructure, e.g., the warmair aisles, the power supply associated with the plurality of computerracks, or the fan coil unit. In some implementations, adjusting anothercharacteristic of the warm air aisles, of the power supply associatedwith the plurality of computer racks, or of the fan coil unit includesadding at least one additional warm air aisle into the data centerbetween the previously existing rows of the additional plurality ofcomputer racks. The associated electrical power density of the computerracks in the data center is reduced based on the additional plurality ofcomputer racks. A dimension of one or more warm air aisles in the datacenter can be reduced based on the reduction of the associatedelectrical power density of the plurality of computer racks in the datacenter. In some implementations, adjusting another characteristic of thewarm air aisles, of the power supply associated with the plurality ofcomputer racks, or of the fan coil unit includes removing a warm airaisle from the previously existing rows of computer racks of the datacenter.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications can be made. For example, variousforms of the flows shown above can be used, with steps re-ordered,added, or removed. Also, although several applications of the coolingsystems and methods have been described, it should be recognized thatnumerous other applications are contemplated. Moreover, although many ofthe embodiments have been described in relation to particular geometricarrangements of cooling and ventilation units, and electronics racks,various other arrangements can also be used. Accordingly, otherembodiments are within the scope of the following claims.

What is claimed is:
 1. A data center, comprising: a plurality of fancoil units operable to circulate a cooling airflow to a human-occupiableworkspace of the data center; a plurality of computer racks arranged ina plurality of rows in the human-occupiable workspace, the plurality ofcomputer racks comprising a plurality of rack-mounted computingequipment that generates heat relative to an electrical power densityassociated with each of the racks, each of the plurality of computerracks comprising an adjustable associated electrical power density; andtwo or more warm air aisles arranged between the plurality of rows ofthe plurality of computer racks, the two or more warm air aisles influid communication with inlets of the fan coil units through at leastone warm air plenum, and also in fluid communication with outlets of thefan coil units through the human-occupiable workspace and the pluralityof computer racks arranged in the plurality of rows, wherein at leastone of the two or more warm air aisles comprises an adjustablecharacteristic based on the adjustable associated electrical powerdensity, and wherein the adjustable characteristic comprises at leastone of a number of the two or more warm air aisles in the data center ora dimension of at least one of the two or more warm air aisles.
 2. Thedata center of claim 1, wherein the adjustable associated electricalpower density comprises: a utilization of the rack-mounted computingequipment; an amount of electrical power supplied to the rack-mountedcomputing equipment; and an amount of cooling airflow supplied to therack-mounted computing equipment.
 3. The data center of claim 1, whereinthe adjustable characteristic further comprises a setpoint temperatureof at least one of the two or more warm air aisles.
 4. The data centerof claim 3, wherein the dimension comprises a width of the at least onewarm air aisle defined between two rows of computer racks.
 5. The datacenter of claim 1, wherein the adjustable characteristic comprises anoperating current of the power supply.
 6. The data center of claim 1,wherein the adjustable characteristic comprises at least one of: anumber of the plurality of fan coil units in the data center; a coolingcapacity of at least one of the plurality of fan coil units in the datacenter; a temperature of the cooling air supplied from at least one ofthe plurality of fan coil units in the data center; and a temperature ofa cooling fluid supplied to at least one of the plurality of fan coilunits in the data center.
 7. The data center of claim 1, furthercomprising: an additional plurality of computer racks arranged in aplurality of rows in the human-occupiable workspace.
 8. The data centerof claim 7, wherein at least one of the two or more warm air aisles, thepower supply associated with the plurality of computer racks, or theplurality of fan coil units comprises an additional adjustablecharacteristic based on the additional plurality of computer racks. 9.The data center of claim 8, wherein the additional adjustablecharacteristic comprises at least one additional warm air aisle into thedata center between the rows of the additional plurality of computerracks.
 10. The data center of claim 9, wherein the associated electricalpower density of the plurality of computer racks in the data centercomprises a reduced associated electrical power density based on theadditional plurality of computer racks, and a dimension of the two ormore warm air aisles in the data center comprises a reduced dimensionbased on the reduced associated electrical power density.
 11. The datacenter of claim 1, wherein each warm air aisle comprises a ducted airwaythat fluidly connects backs of the plurality of computer racks that areopen to the warm air aisle to the warm air plenum.
 12. The data centerof claim 11, wherein the warm air plenum comprises an attic space in thedata center.
 13. The data center of claim 2, wherein the adjustablecharacteristic further comprises a setpoint temperature of at least oneof the two or more warm air aisles.
 14. The data center of claim 13,wherein the dimension comprises a width of the at least one warm airaisle defined between two rows of computer racks.
 15. The data center ofclaim 13, wherein each warm air aisle comprises a ducted airway thatfluidly connects backs of the plurality of computer racks that are opento the warm air aisle to the warm air plenum.
 16. The data center ofclaim 15, wherein the warm air plenum comprises an attic space in thedata center.
 17. The data center of claim 13, further comprising anadditional plurality of computer racks arranged in a plurality of rowsin the human-occupiable workspace.
 18. The data center of claim 17,wherein at least one of the two or more warm air aisles, the powersupply associated with the plurality of computer racks, or the pluralityof fan coil units comprises an additional adjustable characteristicbased on the additional plurality of computer racks.
 19. The data centerof claim 18, wherein the additional adjustable characteristic comprisesat least one additional warm air aisle into the data center between therows of the additional plurality of computer racks.
 20. The data centerof claim 19, wherein the associated electrical power density of theplurality of computer racks in the data center comprises a reducedassociated electrical power density based on the additional plurality ofcomputer racks, and a dimension of the two or more warm air aisles inthe data center comprises a reduced dimension based on the reducedassociated electrical power density.